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ABSTRACT 



The purpose of this compendium is to assist administrators, 
staff, evaluators, and alcohol and other drug (AOD) prevention specialists in 
identifying and selecting feasible, informative, and appropriate evaluation 
instruments to assess AOD use and monitor program effectiveness on campus. 

The use of qualitative methods in the evaluation of grant-funded projects is 
discussed. Section 1, "Choosing the Right Instrument," presents the primary 
issues that must be considered when choosing an instrument, which range from 
format and feasibility to technical concerns about the quality of the 
measure. Section 2, "A Compendium of Measures," has short descriptions of 
seven instruments selected by a panel of experts in AOD prevention. These 
tools are among the best in the field and are designed to cover a variety of 
areas including frequency and amount of AOD use, consequences of AOD use, 
student perceptions of AOD use on campus, fraternity and sorority AOD 
environment, faculty and staff perceptions of AOD use, and community 
coalition involvement in prevention efforts. Section 3, "Annotated Evaluation 
Bibliography," covers resources on research design, statistical analysis, 
evaluation, and other instruments. (EMK) 
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Selecting the Right Tool 

A Compendium of Alcohol and Other Drug Assessment 
and Evaluation Instruments for Use in Higher Education 

Cheryl Presley/S. Bryn Austin/ Judith Jacobs 



This article discusses the use of qualitative methods in the evaluation of 
grant-funded projects. 
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PREFACE 



The Higher Education Center for Alcohol and Other Drug Prevention was 
established by the U.S. Department of Education in 1993 to assist 
institutions of higher education in developing and carrying out alcohol and 
other drug (AOD) prevention programs that will promote campus and community 
safety and help nurture students* academic and social development. 

To accomplish this mission, the Center seeks to increase the capacity of 
postsecondary schools to develop, implement, and evaluate programs and 
policies that are built around environmental management strategies. 
Environmental management means moving beyond general awareness and other 
education programs to identify and change those factors in the physical, 
social, legal, and economic environment that promote or abet alcohol and 
other drug problems. 

Clearly, stemming the use of alcohol and other drugs is not something that 
college administrators alone can achieve. Top administrators, especially 
presidents, must exercise leadership, but their success will depend 
ultimately on their ability to build a strong coalition of both on-campus 
and community interests. The better AOD prevention programs are campuswide 
efforts that involve as many parts of the college as possible, including 
students, staff, and faculty. For this reason, the Center emphasizes 
team-focused training and technical assistance work. 

Building coalitions with local community leaders is also key. College 
campuses do not exist in isolation. AOD prevention planners need to 
collaborate with local leaders to limit student access to alcohol, prevent 
intoxication, and support the efforts of local law enforcement. The Center 
therefore seeks to motivate and train academic leaders to work with local 
community representatives, while also joining with national organizations 
that urge local coalitions to increase their outreach to academic 
institutions . 
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prevention at institutions of higher education. The concepts and approaches 
it describes should be viewed in the broader context of prevention theory 
and the approaches affirmed by the U.S. Department of Education and 
promoted by the Center in its training, technical assistance, publication, 
and evaluation activities. 

For information on Center services, please contact: 

The Higher Education Center for Alcohol and other Drug Prevention 
Education Development Center, Inc. 

55 Chapel Street 

Newton, MA 02458-1060 

Tel.: (800) 676-1730 

Fax: (617) 928-1537 

Website: http://www.edc.org/hec/ 

E-mail: HigherEdCtr0edc.org 

INTRODUCTION 



The aim of this compendium is to assist administrators, staff, evaluators, 
and AOD prevention specialists in identifying and selecting feasible, 
informative, and appropriate evaluation instruments to assess AOD use and 
monitor program effectiveness on campus. 

Since passage of the federal Drug-Free Schools and Communities Act of 1986, 
colleges and universities have been required to maintain alcohol and other 
drug (AOD) prevention programs and policies and to conduct biennial reviews 
of their effectiveness. Partly in response to the federal mandate, 
institutions of higher education across the country have launched a myriad 
of ambitious programs and policy changes. Increasingly, school officials 
have begun to devote a significant amount of staff time and resources to 
systematic, campus-based assessment of AOD use at their individual 
institutions. They recognize that the available national data on college 
AOD use, while informative, are not alone a sufficient basis on which to 
make programming decisions. 

Assessing AOD use and measuring the effects of prevention efforts can be a 
daunting task, especially when there are a number of programs on campus 
that promote prevention on multiple levels. A single institution may have 
in place AOD prevention programs designed to address individual student 
behavior, interpersonal behavior, perceptions of social norms, 
institutional change, and community-level change. High demands on resources 
and staff time also pose challenges to evaluation work. It may be 
unrealistic to expect every campus to conduct a rigorous, experimental 
evaluation of its programming and policies. But even so, all schools need 
to use key indicators to monitor progress and suggest needed modifications. 
This document will help in either case. 



Often the most basic piece of information collected in campus assessments 
is the prevalence of AOD use. The proportion of students drinking alcohol 
or using other drugs, the frequency of use, and the amount of use are all 
critical to understanding the AOD use patterns at a college or university 
and to monitoring program impact. But there are many other factors that 
evaluations can assess to help improve understanding of the AOD climate on 
campus. Selecting the right data collection instruments is a vital 
component of evaluation. This compendium covers the important issues to 
consider when making those selections and identifies the leading 
instruments used in the postsecondary AOD prevention field. The instruments 
included in this compendium not only assess student AOD use but also cover 
consequences of AOD use; student, staff and faculty perceptions of AOD use; 
environmental factors that encourage use (for instance, tailgate parties) 
or discourage use (such as substance-free housing); and other issues 
related to the college or university AOD climate. 



The chief criteria 



O 

ERIC 

2 of 18 



for inclusion of the 



instruments in this 



compendium are 



3 



2/18/99 10:22 AM 



http://www.edc.org/hec/pubs/selecting-right-tool.txt 



that they are 

* designed specifically for a postsecondary student population or 
environment 

* well-tested, valid (a term discussed in section 1 meaning that the 
instrument actually measures what it purports to measure), and 
reliable (meaning that the instrument provides consistent measurements 
over time) 

* intended for general survey research and not for the diagnosis of 
alcoholism or alcohol abuse problems in individuals 

Section 1 of the compendium presents the primary issues that must be 
considered when choosing an instrument, which range from format and 
feasibility to technical concerns about the quality of the measure. 

Section 2 has short descriptions of seven instruments selected by a panel 
of experts in AOD prevention. These tools are among the best available in 
the field and are designed to cover a variety of areas relevant to campus 
AOD prevention, including assessment of 

* frequency and amount of AOD use 

* consequences of AOD use 

* student perceptions of AOD use on campus 

* fraternity and sorority AOD environment 

* faculty and staff perceptions of AOD use 

* community coalition involvement in prevention efforts 

Section 3 contains an annotated bibliography of resources on research 
design, statistical analysis, other aspects of evaluation, and additional 
instruments not included in this compendium. 

While informed and careful instrument selection is central to quality 
evaluation, it is only one piece of a comprehensive .evaluation strategy. It 
is hoped that the information included in this compendium will serve as a 
valuable resource for administrators, staff, evaluators, and prevention 
specialists as they develop their evaluation plans and work to increase the 
effectiveness and efficiency of prevention programming and policy on 
America’s college and university campuses. 



SECTION 1: CHOOSING THE RIGHT INSTRUMENT 
Key Features of Instruments 

* Instruments vary a great deal, not only in terms of the information 
they assess but also in the way they assess it and how well they do 
so. Within the AOD prevention field, instruments have been designed to 
collect data on a range of factors, from student AOD use patterns to 
perceptions of campus norms to environmental conditions that promote 
or deter AOD use. Before choosing an instrument it is important to 
determine what kind of data is needed. If it has not been clearly 
established which questions about AOD use on campus are the most 
pressing, there is a risk of selecting an instrument that focuses on 
irrelevant information or that omits important data. In section 2 of 
this compendium, the domain, or range of factors covered by an 
instrument, is described in the summaries under Application and 
Outcomes . 
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* A second consideration in selecting an instrument is the target 

population. Instruments are designed with a particular group of people 
in mind. For instance, a measure intended for adults may be too 
difficult for children to understand. Also, a measure surveying 
college students about drinking on campus would not be appropriate for 
young adults in the same age range if they were not attending a 

4 



2/18/99 10:22 AM 



http://www.edc.org/hec/pubs/selectmg-right-tool.txt 



college or university. All the instruments presented in this 
compendium are appropriate for the college community, but the intended 
respondents may be students, faculty and staff, or another subgroup on 
campus, depending on the measure. In section 2, the target population 
is described under Informants. 

* A third consideration is the format of the instrument. Many use the 
self-report, forced-choice format typical of standardized tests such 
as the SAT. Others are self-report but use open-ended questions to 
allow respondents more freedom in answering. Still others use an 
interview format, in which a trained staff person conducts a 
semistructured interview with a respondent or a group of respondents 
and records their comments. A self-report, forced-choice format is the 
most feasible for surveying a large number of people but is more 
restricted than the other formats in the detail of information that 
can be gathered. The interview format tends to be more labor intensive 
for staff compared with other survey techniques, but it also has the 
potential to collect more detailed and complex information from 
respondents. Format and length determine the time and effort required 
of respondents and may limit the settings in which an instrument can 
be administered. For instance, if students are to be surveyed during 
class time set aside by a cooperating professor, then a 20-minute, 
forced-choice questionnaire may be most feasible. Information about 
the format of the instruments is detailed in section 2 under Format 
and Administration. 

* A fourth and arguably the most important consideration is the 
technical quality of instruments. Experts in evaluation refer to 
technical quality as the validity and reliability of an instrument. 
Validity is an indication of an instrument's ability to measure what 
it is intended to measure. Reliability indicates the consistency with 
which the instrument provides those measurements. Consider, for 
example-, a ruler, an everyday type of measuring instrument. If a ruler 
has high validity, then we know that the space between lines marking 
an inch truly is one inch. If the ruler has high reliability, it will 
measure one inch as the same amount of distance every time we use the 
ruler . 

Determining validity and reliability is essential to establishing the 
technical quality of an instrument and ultimately to producing credible 
evaluation results. Since validity and reliability testing is time 
consuming, the instrument selection process should always include a careful 
review of existing instruments that have been fully tested. Section 2 of 
this compendium includes information on the validity and reliability of 
each of the featured instruments. Because an understanding of these 
concepts is so critical to well-informed instrument selection, they are 
described in greater detail below. 

Validity 

There are three types of validity used in the field: content validity, 
construct validity, and predictive validity. In general, validity is not an 
all-or-nothing characteristic; rather, it is described in degrees from low 
to high. The more evidence gathered through multiple studies that shows 
that an instrument performs well in terms of the three types of validity, 
the more confidence evaluators can have in the measure and, most important, 
in the study results. 

Content Validity. 

Content validity refers to whether the questions — called items — included 
in an instrument cover the whole domain of factors it is intended to 
address. This form of validity is often determined by agreement among 
experts. For example, an instrument intended to measure patterns of alcohol 
use on a college campus would need to include items covering at least three 
areas: the proportion of students who drink, how frequently they drink, and 
how much they drink. If a measure addressed only one of these aspects of 
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college student alcohol consumption, experts would consider it to have poor 
content validity. 

Construct Validity. 

Construct validity refers to the extent to which an instrument successfully 
measures a theoretical concept-called a construct-such as anxiety, peer 
pressure, or perceptions of drinking norms. Scores from one instrument can 
be compared with scores from others that are intended to measure the same 
construct to assess whether the instruments perform comparably. Moderate or 
high correlation between instruments designed to measure the same construct 
suggests that the instruments are performing as intended. 

Predictive Validity. 

Predictive validity indicates an instrument's ability to provide meaningful 
patterns of results. For instance, student performance on the SAT is used 
by admissions committees to predict academic performance in college. How 
well SAT scores actually do predict success in college would be an 
indication of the SAT's level of predictive validity. Predictive validity 
is also used to refer to an instrument's ability to discriminate among 
groups of respondents who would be expected to score differently on a 
particular measure. For example, a researcher could conduct interviews with 
students about their alcohol consumption and then divide them into two 
groups, one made up of heavy drinkers and the other of light drinkers. If 
the researcher then had the students complete a questionnaire on AOD use, 
the expectation would be that the heavy drinkers would score differently on 
the instrument than light drinkers. If the questionnaire could not 
discriminate between the two groups, it would be considered to have low 
predictive validity. 

Reliability 

Reliability, which refers to a measure's consistency, can be divided into 
three main types: test-retest reliability, alternate forms reliability, and 
internal consistency. 

Test-Retest Reliability. 

An instrument's ability to measure a construct the same way at repeated 
testing times is referred to as its test-retest reliability. To assess this 
type of reliability, an instrument may be administered twice to a group of 
people. If the time in between testing is not too long and if the construct 
being measured is relatively stable, the scores at the first test would be 
expected to be very similar to the scores at the second test. For instance, 
a group of students could be surveyed about their AOD use over the past 
year and surveyed again two weeks later. Their reports of AOD use over the 
past year should not have changed much over the two weeks, so if an 
instrument has a high test-retest reliability, each student's score on the 
first test should highly correlate with his or her score on the second 



The magnitude of the test-retest correlation can be represented by a number 
ranging from 0 to 1, where 1 indicates perfect correlation and 0 represents 
no correlation. In the context of test-retest reliability, a correlation of 
.85 or above is generally considered high. 

Alternate Forms Reliability. 

Reliability can also be established if two alternate forms of the 
instrument produce highly correlated scores when administered to the same 
group of people. The alternate forms of the instrument must be comparable, 
that is they must include the same types of questions in the same format. 
Evaluators may want to use more than one version of a questionnaire in a 
situation, for instance, where surveys are repeated in a short time period, 
raising concern that respondents' memory of their answers may influence 
subsequent responses. 

Internal Consistency. 

Internal consistency is described by the correlations among items within an 
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instrument. If an instrument measures a single construct, all the items 
would be expected to be highly correlated. Likewise, if an instrument is 
really measuring two related but distinct constructs, the items might 
cluster into two groups, where those relating to one construct correlate 
highly with each other but less so with items in the other group and vice 
versa. For instance, if an instrument included questions about both binge 
drinking and consumption of alcohol at meals, scores on all the items about 
binge drinking would be expected to be highly correlated with each other 
but perhaps only moderately correlated with items on drinking at meals. 

Internal consistency is frequently represented by a statistic called 
Cronbach*s alpha, which also can range from 0 to 1. Cronbach*s alpha above 
.85 is considered high but a value above .70 usually is considered an 
indication of adequate internal consistency. Another number used to 
represent internal consistency is the item-to-total correlation, which 
indicates the correlation — again, ranging from 0 to 1 — between the score 
on any one item and the total score on the test as a whole. Generally, 
scores for each item should fall between .3 and .7. 



SECTION 2: A COMPENDIUM OF MEASURES 

Alcohol and Other Drug Surveys 
Core Alcohol and Drug Survey 

Key Contact: 

Core Institute 
Student Health Programs 
Kesnar Hall MC 6802 
Southern Illinois University 
Carbondale, IL 62901 
Phone: (618) 453-4366 

Website : http : //www. siu . edu/ --co reins t/ 

Application and Outcomes: 

The Core Alcohol and Drug Survey was developed in 1989 to assess the 
nature, scope, and consequences of alcohol and other drug use on college 
campuses. This survey was expanded in 1994 to include assessment of 
students’ perceptions and beliefs about alcohol and other drug (AOD) use 
and AOD- related sexual behavior and violence. 

For each behavior measured, an item score can be obtained. Quantity and 
frequency of use is measured by annual prevalence of use for all drugs and 
alcohol, 30-day use for all drugs and alcohol, average number of drinks per 
week, binge drinking within the last two weeks, and change in drinking and 
drug use in the past 12 months. Problem-related questions measure 19 
consequences of drinking and drug use by frequency of occurrence. Another 
set of questions measures AOD-related incidents of violence. Several items 
also measure perception of risk, perception of others' use, perceptions of 
others' feelings regarding use, and perception of campus environment. 

Social beliefs about the effects of alcohol are measured by 14 items using 
a forced-choice yes/no format. There are questions that assess other campus 
variables, such as the extent of participation in campus activities, 
frequency of refusal behavior, and interference from others' drinking in a 
student's life on or around campus. 

Informants : 

Postsecondary student population 
Format and administration: 

The long-form version of the self-report Core Alcohol and Drug Survey 
includes 39 questions on use and consequences, as well as demographics. The 
short-form version included 23 questions. Individual questions have between 
2 and 19 response options. The most common methods of administration are to 
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mail surveys to a randomly selected subset of students or to administer 
surveys in randomly selected classrooms. Each method is discussed in the 
Core Survey User's Manual, which is distributed upon purchase of the 
instrument. Colleges can have the results of the survey analyzed by the 
Core Institute. The long form of the survey takes from 20 to 35 minutes for 
respondents to complete. 

Technical Assistance: 

Technical assistance for those administering the Core Alcohol and Drug 
Survey is available from the Core Institute through a 15-minute video ($25) 
that describes how to conduct the survey; through the Core Institute's 
Website (http : //www. siu . edu/departments/coreinst/public_html/index . html ) ; or 
via telephone. 

Costs : 

The Core Alcohol and Drug Survey is distributed from and analyzed by the 
Core Institute. The costs for obtaining the long form of the survey and 
having the results compiled and analyzed are as follows: 

Purchase of each survey copy: $ 0.22 
Scanning of each returned survey: $ 0.17 
Executive Summary : $20 . 00 
Abbreviated Analysis : $45 . 00 
Disk with Raw Data: $13. 00 

Use in Evaluation: 

The Core Alcohol and Drug Survey is one of the most rigorously developed 
survey instruments for postsecondary populations. Its original intent was 
to assess only the frequency and consequences of AOD use. After hundreds of 
two- and four-year institutions of higher education had used the Core 
Survey, many requests were received to expand the content of the survey to 
include other aspects of campus life related to substance issues. 
Consequently, the survey was expanded to include questions on sexuality, 
campus violence, institutional climate, perceptions of AOD use, and 
extracurricular activity involvement. Development of both the original and 
expanded Core Survey followed strict American Psychological Association 
(APA) guidelines for test development. 

This instrument is not used to diagnose alcohol dependency in individuals 
but rather to assess the level and impact of alcohol and other drug use on 
campus. It is a valuable tool for determining how to target populations for 
prevention programming, designing social marketing and media advocacy 
campaigns, and assessing the impact of these prevention efforts. 

Validity and Reliability: 

Considerable evidence supports the validity of the Core Survey. To 
establish content validity, existing instruments and literature were 
reviewed to ensure that important aspects and consequences of alcohol and 
other drug use were adequately covered. A panel of experts then reviewed 
the items to assess whether they sampled the domain of interest. The level 
of agreement for item inclusion among the experts was very high (interrater 
reliability was .90). 

Test-retest reliability was estimated using Pearson product-moment 
correlation coefficients. For items on AOD use and consequences of use, 
test-retest reliability was high, with the majority of item correlations 
falling above .80. Items on campus AOD norms showed moderate test-retest 
reliability, with most correlations falling between .30 and .80. 
Item-to-total correlations were calculated to assess the internal 
consistency of the survey. Correlations for the majority of items on AOD 
use, consequences, and campus norms fell between .30 and .70, as 
recommended by Henryssen. 

Comments : 

The Core Alcohol and Drug Survey is easy to administer and has substantial 
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validity and reliability data to support its use with postsecondary 
populations. Optical scan scoring of the survey is done at the Core 
Institute, and a report of results is provided to the institution 
administering the survey, generally within a week. In addition, the Core 
Institute can provide a computer disk with the institution's survey data in 
a format that will allow further data analysis. 

Another benefit of using the Core Survey is that institutions can compare 
results with those of similar institutions from across the country that 
have also used the instrument. Compiled results from other colleges and 
universities are published in Core Institute monographs every two years and 
are available on the Core Institute's Website. 



Alcohol and Other Drug Surveys 

Campus Survey of Alcohol and Other Drug Norms* 



Key Contact: 

Core Institute 
Student Health Programs 
Kesnar Hall MC 6802 
Southern Illinois University 
Carbondale, IL 62901 
Phone: (618) 453-4366 

Website : http : //www. siu . edu/'-coreinst/ 

Applications and Outcomes: 

The Campus Survey of Alcohol and Other Drug Norms was developed in 1997 to 
assess respondents' AOD use and their perceptions of campus norms and 
substance use by other students. 

The survey was developed in response to the almost universal finding that 
students overestimate AOD use by their peers. This misperception is of 
concern to the extent that students then try to emulate the perceived norm. 
The first set of questions asks how frequently the respondent believes 
students at their institution use alcohol and marijuana and other illicit 
drugs. The second set of questions asks students to estimate the percentage 
of students who do not drink alcohol and of those who binge drink. Students 
are asked to report their perceptions of how much students typically 
consume and of places where alcohol is consumed. There is also a section 
that elicits respondent attitudes regarding other aspects of drinking and 
drug use. Finally, respondents are asked about their perceptions of campus 
policy and policy enforcement. 

Informants : 

Postsecondary student population 



Format and Administration: 

The self-report Campus Survey of Alcohol and Other Drug Norms consists of 
17 behavioral, perceptual, and attitudinal questions and nine demographic 
questions. Items range from 2 to 10 response options. Respondents average 
12 minutes to complete the survey. The campus survey can be combined with 
the Core Alcohol and Drug Survey or the Faculty and Staff Environmental 
Alcohol and Other Drug Survey to elicit a broad view of AOD problems on the 
campus. The Core Institute is available to optically scan questionnaires 
and analyze survey results for individual colleges and universities. 



Technical Assistance: 

Technical assistance for those administering the Campus Survey of Alcohol 
and Other Drug Norms is available from the Core Institute through a 
15-minute video ($25) that describes how to conduct the survey; through the 
Core Institute's Website 

(http : //www. siu. edu/departments/coreinst/public_html/ index . html ) ; or via 
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telephone . 

Costs : 

The Campus Survey of Alcohol and Other Drug Norms is distributed from and 
analyzed by the Core Institute. The costs for obtaining the survey and 
having the results compiled and analyzed are as follows: 

Purchase of each copy: $ 0.27 

Scanning of each returned survey: $ 0.17 

Executive Summary: $25.00 

Cross-Tab Analysis: $30.00 

Disk with Raw Data: $13.00 

Use in Evaluation: 

While the Campus Survey of Alcohol and Other Drug Norms was not introduced 
on college campuses until 1997, various items on the survey have been used 
extensively in research and to develop social marketing programs for 
correcting campus misperceptions regarding AOD use and changing campus 
norms. The items were based on theory in AOD prevention research. 

Validity and Reliability: 

Content validity was assessed based on the level of agreement among the 
experts for item content. There was unanimous agreement among the experts 
on the choice of items. Many of the items have been evaluated for use in 
other instruments. 



The campus survey has a strong basis in theory, and many of its items have 
been tested in earlier research studies. In addition, results were found to 
be stable over a four-week test-retest period. The survey's design enhances 
ease of administration, scoring, and interpretation of the results. 



Environmental Assessment 

Faculty and Staff Environmental Alcohol and Other Drug Survey 

Key Contact: 

Core Institute 
Student Health Programs 
Kesnar Hall MC 6802 
Southern Illinois University 
Carbondale, IL 62901 
Phone: (618) 453-4366 

Website : http : //www. siu . edu/--coreinst/ 

Application and Outcomes: 

The Faculty and Staff Environmental Alcohol and Other Drug Survey, 
developed in 1993, consists of five subscales developed to assess faculty 
and staff perceptions of AOD problems on campus, awareness of policy and 
policy enforcement, support for programming efforts to combat AOD problems, 
awareness of university assessment efforts, and faculty and staff's 
perceived ability to identify students who are experiencing a problem and 
refer them for help. The basis for this instrument was the assumption that 
faculty and staff attitudes, perceptions, and beliefs about the extent of 
alcohol and other drug problems are a major influence on the campus 
environment . 

Informants : 

Postsecondary faculty and staff 
Format and Administration: 

The Faculty and Staff Environmental Alcohol and Other Drug Survey consists 
of 35 items in a forced-choice format, plus seven demographic questions. 



Comments : 
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Responses are marked on an optical scan sheet. The Core Institute is 
available to analyze survey results for individual colleges and 
universities. The survey usually takes between 20 to 30 minutes to 
complete . 

Technical Assistance: 

Technical assistance for those administering the Faculty and Staff 
Environmental Alcohol and Other Drug Survey is available from the Core 
Institute through a 15-minute video ($25) that describes how to conduct the 
survey; through the Core Institute's Website 

(http://www.siu.edu/departments/coreinst/public_html/index.html) ; or via 
telephone . 

Costs : 

The Faculty and Staff Environmental Alcohol and Other Drug Survey is 
distributed from and analyzed by the Core Institute. 

The costs for obtaining the long form of the survey and having the results 
compiled and analyzed are as follows: 

Purchase of each copy: $ 0.19 
Scanning of each returned survey: $ 0.13 
Cross-Tab Analysis: $30.00 
Disk with Raw Data: $13.00 

Use in Evaluation: 

The faculty and staff survey has been used in faculty workshops and 
trainings, pre- and posttest evaluations, preintervention assessments, and 
in the development of media advocacy, social norms, and social marketing 
programs. The survey provides information about faculty and staff attitudes 
and their willingness to support prevention programming efforts. 
Additionally, this instrument can serve as a recruitment tool for faculty 
-and staff involvement in campus efforts to prevent alcohol and other drug 
problems . 

Validity and Reliability: 

Research supports the validity of the faculty and staff survey. The level 
of agreement among the experts on the instrument was .90 in an evaluation 
of content validity. Some of the items included in the instrument were 
validated using other instruments such as the Core survey and the National 
Forum Assessment Guide. The item-to-test correlation is strong, giving more 
evidence of the instrument's validity. 

Cronbach's alpha for the instrument as a whole is .81. Cronbach' s alpha for 
the five subscales are: policy =.75, prevention program awareness = .65, 
involvement in prevention = .67, assessment = .57, and perception of 
problem = . 62 . 



The faculty and staff survey can be used in a number of ways to provide a 
more complete understanding of the total campus environment. As one of the 
most stable elements in the college community, faculty and staff can 
provide valuable insight into factors affecting student life and the whole 
campus environment. 



Environmental Assessment 

College Alcohol Risk Assessment Guide 

Key Contact: 

The Higher Education Center for Alcohol and Other Drug Prevention 
55 Chapel Street 
Newton, MA 02458-1060 



Comments : 
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Phone: (617) 969-7100; Toll-free: (800) 676-1730 

Website: http://www.edc.org/hec/ 

Application and Outcomes: 

The College Alcohol Risk Assessment Guide (CARA) was developed in 1994 and 
revised in 1997. This instrument provides campus teams with tools for 
systematic investigation of the extent of alcohol problems on and off 
campus and for assessment of the environment that shapes student drinking 
decisions and adverse consequences. Campuses are encouraged to use the CARA 
as part of broad-based campus prevention initiatives that involve 
coalitions and task forces. Unlike previous instruments reviewed in this 
handbook, CARA does not provide tools for tracking quantitative data 
regarding alcohol use and consequences. Therefore, the authors recommend 
the use of CARA in combination with other survey instruments such as the 
Core Survey. 

Informants : 

Postsecondary students, faculty, staff, including groundskeepers and 
housekeeping staff, and other members of the campus and surrounding 
community 

Format and Administration: 

CARA is a narrative publication that provides a set of tools to assist 
campuses in understanding environmental approaches to prevention and in 
collecting information in a structured way in order to describe and monitor 
environmental factors in alcohol use. CARAis best used within the context 
of a prevention coalition or task force. These groups can pick and choose 
the tools that will most assist them in understanding and implementing 
environmental prevention measures. 

Technical Assistance: 

Technical assistance is available through the Higher Education Center for 
Alcohol and Other Drug Prevention. Center staff and trained Center 
Associates are available to answer questions regarding the instrument and 
to offer guidance during its administration. 

Costs: CARA is available free from the Higher Education Center. 

Use in Evaluation: 

CARA analysis tools can be used to collect baseline information for later 
comparison to postintervention data to assess prevention program impact. 
Ongoing collection of information on environmental factors using CARA tools 
will enhance a long-term program evaluation. 

Validity and Reliability: 

CARA assessment technology is based upon the use of social indicators, 
which have a long history of acceptance in economics, public health, and 
many other fields. These tools collect primarily qualitative data and other 
information useful in program planning and program monitoring. Though not 
standardized instruments, CARA tools do provide a structured way to collect 
data . 

Comments : 

The development of CARA occurred over a three-year period. During this time 
focus groups, field test groups, a pilot test, and dissemination sessions 
were held. Over 300 people were trained in the use of these techniques and 
tools. As shown in a follow-up evaluation, most who were involved in 
trainings believed the information was valuable to them as prevention 
specialists . 

CARA is aimed at a wide audience on campuses to promote an understanding of 
the importance of information-based decision making and planning in AOD 
prevention. The guide’s main goal is to shift the focus of campus attention 
from individuals to the broader environment. 
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Environmental Assessment 
Community Coalition Surveys 
Key Contact: 

Center for Alcohol and Addiction Studies 
Box G-BH 

Brown University 
Providence, RI 02912 
Phone: (401) 444-1807 

Attn: Roger Mitchell, Ph.D. 
e-mail: Roger_Mitchell0Brown.edu 

Application and Outcomes: 

Community coalitions are sometimes organized to support primary prevention 
of AOD problems on college and university campuses. Several instruments 
designed by the Community Research and Services Team at the Center for 
Alcohol and Addiction Studies at Brown University could be revised for use 
in campus -community prevention programming. These instruments, designed to 
assess resources, efforts, and efficacy of community prevention coalitions, 
include three surveys developed in 1989 that are directed at different 
perspectives on community coalitions: coalition members, coalition leaders, 
and coalition steering committee members. Additionally, coalition-generated 
prevention plans can be coded to examine the specifics of planned 
prevention activities. 

The Coalition Member Survey assesses the level of involvement by coalition 
members, costs and benefits of participation, perceptions of individual 
skills, perceptions of the efficacy of the coalition, and beliefs about the 
resulting effects of the coalition members' contributions. 

The Coalition Leader Interview assesses perceptions of the strengths and 
weaknesses of coalition efforts. It assesses satisfaction with current 
membership, the organizational structure, resources, contacts with other 
agencies and coalitions, perceived success, and technical assistance needs. 

The Executive (or Steering) Committee Member Survey assesses perceptions of 
the goals and processes for the coalition, perceived benefits of 
participating in the coalition, difficulties associated with participation, 
and perceptions of the coalition' s governance and decision-making 
structure . 

The Typology of Prevention Activities provides content categories (e.g. 
raising awareness, skill building, changing organizational policies) into 
which trained coders can reliably classify planned programming. This 
classification scheme can then be used to assess the scope and 
appropriateness of the types of prevention activities that comprise the 
programming as a whole. 

Informants : 

Community coalition members and leaders 
Format and Administration: 

The Coalition Member Survey is a six-to-eight page, forced-choice, 
self-report instrument that takes 20 to 25 minutes to complete. It has 
typically been administered by mail. The Coalition Leader Interview is an 
in-depth, 16-20 page interview with both forced-choice and open-ended 
response options, which allows the interviewer to probe as needed. The 
interview takes about 45 minutes to complete. (A shorter version can be 
conducted by telephone.) The self-report Executive Committee Member Survey 
uses a forced-choice response format and takes just 10 to 15 minutes to 
complete . 



er|c 



12 of 18 



13 

2/18/99 11:38 AM 



http://www.edc.org/hec/pubs/selecting-right-tool, txt 



The Coalition Member Survey and Executive Committee Member Survey require 
written responses. They cannot be scanned optically, but the response 
options for most items easily lend themselves to coding for computer 
analyses. Responses to the Coalition Leader Interview are recorded by the 
interviewer, and some open-ended questions require interpretation and 
coding. For each of the above surveys, subscales can be added or deleted 
depending upon the interests of the evaluator. 

With the typology measure, each prevention activity is coded in one of 
several specific categories. This allows users to construct a profile that 
graphically displays where coalitions are devoting their efforts. An 
additional measure can be calculated to assess the comprehensiveness and 
scope of the prevention strategies being used. 

Technical Assistance: 

While no formal technical assistance currently exists, several articles 
describing how the instruments may be used are available {see footnotes to 
these pages) . In addition, enquiries may be directed to Dr. Mitchell at 
Brown University. 



No costs are associated with obtaining the survey instruments. However, the 
Center for Alcohol and Addiction Studies asks that acknowledgment be given 
and that outcomes and reports generated through the use of the instruments 
are shared with the Center. 

Use in Evaluation: 

The constructs measured by these surveys can be used to assess the efficacy 
of many types of coalitions. Since coalitions are often made up of 
individuals holding disparate views, the results of these surveys may 
provide excellent information for planning, monitoring, and redirecting 
coalition development. 

validity and Reliability: 

Individual scales from the instruments have undergone psychometric 
development, including principal components analysis and calculation of 
Cronbach's alphas and correlations with other scales to investigate 
construct validity. Cronbach's alphas on most subscales are above .80. 

For the Typology of Prevention Activities, the level of agreement among the 
experts on reliability in assigning activities to categories was at 
adequate levels {Cohen's Kappa = .65). In addition, a measure of the scope 
or comprehensiveness of coalition work was created based on the number of 
different types of strategies {such as raising awareness of a problem or 
skill building) used. When comprehensiveness scores were compared to 
independent ratings by staff, the correlation was r = .51, suggesting 
moderate convergent validity. 

Comments : 

These surveys ask important questions that can be used to enhance the 
efficacy of task forces and coalitions. The information can help close 
information gaps and address issues of disempowerment and discouragement 
among task force members. These surveys are appropriate for use in college 
communities seeking environmental change in AOD use. 



Fraternity and Sorority Assessment 

The Greek Experience: A Survey of Fraternities and Sororities 
Key Contact: 

Office of the Vice President for Student Development and Athletics 
110 Morrill Hall 
University of Minnesota 



Costs 
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100 Church Street, SE 
Minneapolis, MN 55455 
Phone: (612) 626-8127 

Attn: Roger Harrold, Ph.D. 

Application and Outcomes: 

Originally a 1986 University of Minnesota survey, the Greek Experience: A 
Survey of Fraternities and Sororities was adapted in 1996 to be usable at 
any college or university. It assesses the problems, attitudes, behaviors, 
and basic demographics of sorority and fraternity members. This 
comprehensive instrument also assesses many aspects of the Greek 
experience, including chapter affairs, academics, personal growth, and 
alcohol use. Several questions from the Core Alcohol and Drug Survey are 
used to assess alcohol-related behavior problems. Six questions tap into 
personal growth, development, and involvement. Revisions to the survey will 
be completed with the input of national Greek leaders and scholars who will 
make suggestions for refining the survey. No changes will be made to the 
Core questions. 

Informants : 

College sorority and fraternity members 
Format and Administration: 

The Greek survey is a 63-item, self-report instrument with a mix of 
forced-choice and open-ended questions. There are 8 items on the survey 
assessing problems with alcohol. The survey can be administered through 
mail or in person and takes about 30 minutes to complete. 

Technical Assistance: 

Technical assistance for administering the survey may be obtained by 
telephone or through a manual describing how to conduct the survey on 
campus. In addition, on-site technical assistance in which a Greek adviser 
may assist schools in planning the administration of the survey will soon 
be available. 

Costs : 

The Greek survey is distributed through the Office of the Vice President 
for Student Development and Athletics and analyzed by Minnesota's Office of 
Measurement. Costs associated with the survey are as follows: 

Purchase of each copy: $ 0.35 

Slicing and optical scanning per survey: $ 0.28 
Cross-tabulations report (1,400 tables ): $35 . 00 
Operations and systems costs (including disc 
with raw data):$70.00 

Optional reports on each individual chapter : $30 . 00 
Use in Evaluation: 

Many in the campus community, including fraternity and sorority members, 
have expressed concern about the high use of alcohol by Greek-affiliated 
students. This instrument can provide good baseline data on fraternities 
and sororities to aid prevention programming efforts. 

Validity and Reliability: 

Questions relating to alcohol for this instrument were taken from the Core 
Alcohol and Drug Survey, which has demonstrated high validity and 
reliability. (See above section on the Core Alcohol and Drug Survey. ) The 
level of agreement among the experts for some of the other scales in the 
Greek survey is .90, suggesting high content validity. 

Comments : 

This survey was developed by Dr. Roger Harrold, who has nearly 30 years' 
experience in test development and assessment at the University of 
Minnesota. Some of the items included in this instrument have been used in 
longitudinal studies at that institution for as many as 25 years. 
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An advantage of this survey is the wide range of questions, which cover 
aspects of the Greek experience left untapped by many other instruments. 
Ease of administration and scoring is also a benefit. 



Other Instruments 

Program Evaluation Handbook: Drug Abuse Education 
Key Contact: 

lOX Assessment Associates 
5301 Beethoven Street 
Suite 190 

Los Angeles, CA 90066 
Phone: (310) 822-3275 

Applications and Outcomes: 

The Program Evaluation Handbook: Drug Abuse Education was developed in 1988 
by lOX Assessment Associates for the Centers for Disease Control and 
Prevention to assist evaluators of health education programs dealing with 
alcohol and other drug problems in schools and communities. It presents 
several measures developed by the authors that may be helpful in assessing 
various aspects of program effectiveness not otherwise covered in this 
compendium. 

The Friends and Family Survey assesses the perception of support and 
assistance from those who are closest to the respondent. This can be used 
in a pre- and posttest situation where there is an intervention 
specifically designed to change the respondents* perceptions of their 
social support network. The Systematic Decision Making Survey assesses the 
respondents* ability to identify steps in a systematic decision-making 
process. The Ways of Coping Survey measures the respondents' choices for 
activities in coping with a variety of difficult situations. 

Informants : 

Adults 

Format and Administration: 

These self-report surveys contain from 10 to 20 items with a mix of closed- 
and open-ended response options. Formulas for scoring are provided with the 
surveys, so the administrator can score the results easily. The surveys can 
be administered in a number of settings, including training sessions and 
prevention programs. 

Technical Assistance: 

Technical assistance is not currently available. 

Costs : 

The Program Evaluation Handbook: Drug Abuse Education costs $19.95 plus 
$5.00 shipping and handling. 

Use in Evaluation: 

The surveys are designed for evaluation of drug abuse education programs. 
They are varied and may be used over time to measure changes in specific 
types of program outcomes. 

Validity and Reliability: 

These instruments were developed by experts, but there is no data on the 
technical quality of the measures. 

Comments : 

A limitation is the lack of , reliability and validity data on these surveys; 
however, the authors of the handbook take the reader through the methods 
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for doing small-scale reliability and validity studies that can be 
conducted on college and university campuses. 

ANNOTATED EVALUATION BIBLIOGRAPHY 

Evaluation Resources Available Through the Higher Education Center for 
Alcohol and Other Drug Prevention 

Austin, B. A College Case Study: A Supplement to Understanding Evaluation: 
The Way to Better Prevention Programs. Washington, D.C.: U.S. Department of 
Education, Higher Education Center for Alcohol and Other Drug Prevention, 
1997. 

In telling the story of a fictitious college, this case study helps readers 
get a feel for what is involved in setting up an evaluation of a college 
AOD prevention program, and what can be gained from such a process. This 
publication was prepared by the Center as a companion to Understanding 
Evaluation, which is directed at a broader audience. 24 pp. 

DeJong, W., and Langenbahn, S. Setting and Improving Policies for Reducing 
Alcohol and Other Drug Problems on Campus: A Guide for Administrators. 
Washington, D.C.: U.S. Department of Education, Higher Education Center for 
Alcohol and Other Drug Prevention, 1995, reprinted 1997. 

Written for administrators at both two-year and four-year institutions of 
higher education. Setting and Improving Policies for Reducing Alcohol and 
Other Drug Problems on Campus is especially valuable to the committee 
assigned to develop and revise alcohol and other drug prevention policies. 
Other members of the college community who have a special interest in 
substance use prevention-faculty, students, program directors, campus, 
security, and other staff-will also find this publication a useful 
introduction to campus policy setting. 

The guide outlines a step-by-step process for establishing or revising 
policies to prevent and deal with problems arising from student use of 
alcohol and other drugs on campus. It describes the environmental 
management approach that can be used to create a safer campus that nurtures 
students' academic and social development. 114 pp. 

DeJong, W., and Wechsler, H. Preventing Alcohol-Related Problems on Campus: 
Methods for Assessing Student Use of Alcohol and Other Drugs. Washington, 
D.C.: U.S. Department of Education, Higher Education Center for Alcohol and 
Other Drug Prevention, 1995. 

To develop effective programs and policies that can reduce alcohol-related 
problems on campus, college administrators need to understand fully the 
nature and extent of these problems at their school. This understanding can 
be achieved only if administrators have credible data on patterns of 
student alcohol consumption and drinking-related risk behavior. The best 
way to obtain these data is to conduct an annual survey using a random 
selection of student respondents. 

This guide offers a straightforward method for gathering and interpreting 
student survey data on alcohol-related problems. The procedure is based on 
the methodology used in a national college alcohol study conducted in 1993 
by the Harvard School of Public Health; it can be easily adapted for use on 
all campuses. 48 pp. 

Muraskin, L. D. Understanding Evaluation: The Way to Better Prevention 
Programs, Washington, D.C.: U.S. Department of Education, 1993. 

Developed with funding from the U.S. Department of Education, Understanding 
Evaluation describes the how and why of program evaluation and outlines the 
steps involved, working from the premise that many useful evaluations can 
be conducted by program staff who may not have formal training in 
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evaluation. The information will be valuable to those who need to conduct 
evaluations required under the Drug-Free Schools and Communities Act 
(DFSCA) , as well as those who simply want to find out whether their 
prevention efforts are yielding results. 98 pp. 

Ryan, B. E.; Colthurst, T.; and Segars, L. College Alcohol Risk Assessment 
Guide: Environmental Approaches to Prevention. Washington, D.C.: U.S. 
Department of Education, Higher Education Center for Alcohol and Other Drug 
Prevention, 1994, revised 1997. 

Designed to assist college administrators in identifying and changing 
factors within the campus environment that contribute to alcohol-related 
problems. Factors are examined within the context of the public health 
approach, which emphasizes ways in which the environment shapes behavior. A 
series of exercises based on the SAFIA (scanning, analysis, response, 
assessment) method are given to assist in identifying specific problems at 
a given college. 104 pp. 



Other Evaluation Resources 



Allen, M. J., and Yen, W. M. Introduction to Measurement Theory. Monterey, 
Calif. : Brooks/Cole Publishing Company, 1979. 

This is an excellent reference book for people who are constructing or 
using measures. 310 pp. 

American Psychological Association. Ethical Principles in the Conduct of 
Research with Human Participants. Washington, D.C.: American Psychological 
Association, 1982. 



This volume is designed to help researchers understand the rationale and 
guidelines for using human subjects in research. It is especially useful 
for handling the ethical issues involved in studying high risk behaviors, 
such as alcohol and other drug use. 76 pp . 

Center for Substance Abuse Prevention. Community Risk Assessment Guide: 

Teen Drinking Prevention Program. Rockville, Md. : U.S. Department of Health 
and Human Services, Center for Substance Abuse Prevention, 1995. [NCADI: 
Inventory Number PHD703] 

Methods for conducting a community needs assessment to gather information 
about alcohol use by underage individuals and the community factors that 
influence this us are described. 38 pp. 



Committee for Developing Standards for Educational and Psychological 
Testing. Standards for Educational and Psychological Testing. Washington, 
D.C.: American Psychological Association, 1985. 



This book presents standards for construction of tests and is a helpful 
guide in understanding the ethical issues involved. It is also useful when 
using a survey for which standard norms have not been established. 99 pp. 

Cronbach, L. J., and Furby, L. 1970. How Should We Measure ”Change"-Or 
Should We? Psychological Bulletin 74(1): 68—80. 

This thought-provoking article highlights some of the questions researchers 
face when measuring change and interpreting change scores. 



Fetterman, D. M. ; Kaftarian, S. J.; and Wandersman, A. (eds.). Empowerment 
Evaluation: Knowledge and Tools for Self-Assessment and Accountability. 
Thousand Oaks, Calif.: Sage Publications, Inc., 1996. 
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findings to improve programs and foster self-determination among program 
participants, is discussed. 411 pp . 

Goldfield, N. G., and Pine, M. Measuring and Managing Health Care Quality: 
Procedures, Techniques, and Protocols. Gaithersburg, Md.: Aspen Publishers, 
1996. 

This is a fairly detailed but very readable and timely handbook addressing 
the complexities of process evaluation, continuous program improvement, and 
community involvement. 213 pp . 

Kumpfer, K. L.; Shur, G. H. ; Ross, J. G. ; Bunnell, K. K. ; Librett, J. J.; 
and Millward, A. R. CSAP Technical Report 8: Measurements in Prevention: A 
Manual to Selecting and Using Instruments to Evaluate Prevention Programs. 
Rockville, Md. : U.S. Department of Health and Human Services, Center for 
Substance Abuse Prevention, 1993. [NCADI: Inventory Number BK213] 

Guidelines for identifying measures and instruments for the evaluation of 
an alcohol and other drug prevention program that are relevant to both the 
program and the target population are discussed. 204 pp . 

Linney, J. A., and Wandersman, A. Prevention Plus III: Assessing Alcohol 
and Other Drug Prevention Programs at the School and Community Level: A 
Four-Step Guide to Useful Program Assessment. Rockville, Md. : U.S. 
Department of Health and Human Services, Office for Substance Abuse 
Prevention, 1991. [NCADI: Inventory Number BK188] 

A step-by-step approach to assessing alcohol and other drug prevention 
programs at the school and community level is described. 461 pp . 

Mills-Novoa, B. "The Use of Qualitative Methods in the Evaluation of 
Grant-Funded Projects," in The Grantseeker * s Guide to Project Evaluation, 
Second Edition, edited by Jacqueline Ferguson. Alexandria, Va.: Capitol 
Publications, 1997: 63—69. 
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